Custom Threshold Alerts

Management Console

Custom Threshold Alert (CTA) is a feature designed to assist program managers in measuring and collecting metrics to create Admin reports that provide users with information on how Resolve Actions Pro is functioning at any given time period.
Actions Pro RSMgmt component supports and collects a wide variety of metrics. These metrics are grouped together into several categories;

• Java virtual machine (JVM) statistics
• database statistics
• server load statistics
• Runbook execution statistics.
• The feature is designed to assist Actions Pro administrators.

Central MCP¹ is a separate system for monitoring and administering Actions Pro instances using a GUI. Its function is to:

• Receive and manage alerts
• Take action on a host/instance
• Assist users with more complex operations such as Capacity Planning, Disaster Recovery, and Upgrade.

When a metric data value exceeds a threshold value, a notification is sent to take precautionary measures to stabilize the system. For example, a metric threshold can be defined such that when the memory available in the JVM drops below a threshold value of 10% of total memory, the user is notified to increase the amount of memory.

CTA Configuration

• MCP can monitor and administer any Actions Pro instance. However, it requires information about the instance. User needs to supply:
• CTA with the host information and proper SSH credential of a Actions Pro host
• Adequate privileges to the Actions Pro user to perform the necessary actions

note

The Blueprint files hold the configuration details for a particular Actions Pro host.

Host Organization

Multiple Actions Pro components compose a Actions Pro instance.

• One or more Actions Pro instance(s) can run on a host.
• One or more hosts can form a cluster.
• A cluster can be labeled with a group.

User Management Concepts

CENTRAL MCP users are separated from the MANAGED Actions Pro instances users. Central MCP that can monitor and operate on a host.

• CENTRAL MCP - Actions Pro instance monitors other Actions Pro instances. If a CENTRAL Actions Pro instance is monitoring another Actions Pro instance, then its rules will be "local" to itself but "global" to the monitored Actions Pro.
• MANAGED MCP - Actions Pro instance monitored by other Actions Pro instances (managed by CENTRAL MCP). Threshold rules deployed to monitor other remote Actions Pro instances are called "global" rules.
  • When a cluster/host wants to be monitored, it will configure its blueprint files that specify the master Actions Pro, its cluster name, hostname, and a flag indicating whether the host is to be monitored.  In general, all hosts within this cluster should be registered as a whole.
  • When a cluster/host wants to be removed from monitoring, it will deregister.
• Locally Managed - Actions Pro instance not managed by any other Actions Pro instance.
  • Rules Created Locally: The threshold rule created by a Actions Pro instance for itself is called "local" rules. Local
  • Threshold Rules are rules against thresholds value that has only local scope, meaning that they are created in/ imported into a single cluster and take effect only against this cluster.
  • If all conditions of a rule are met, it will trigger the action specified by the user.

Procedures

The following procedures are used to set up MCP mode in blueprint configuration.

Setting MCP Modes

Actions Pro can build up and organize the collection of cluster/host/instances that it needs to manage.

1. When a cluster/host wants to be monitored, it will configure its mode in its blueprint file example given below.
   • Locally Managed - Actions Pro instance not managed by any other Actions Pro instance. Cluster MCP running mode - options are LOCAL_MODE_MCP (default), MANAGED_MODE_MCP, CENTRAL_MODE_MCP.

     MCPMODE=MANAGED\_MODE\_MCP
   • MANAGED MCP Actions Pro instance monitored by other Actions Pro instances (managed by CENTRAL MCP). Cluster MCP running mode - options are LOCAL_MODE_MCP (default), MANAGED_MODE_MCP, CENTRAL_MODE_MCP.

     MCPMODE=CENTRAL\_MODE\_MCP
   • CENTRAL MCP Actions Pro instance monitor other Actions Pro instances. Cluster MCP running mode - options are LOCAL_MODE_MCP (default), MANAGED_MODE_MCP, CENTRAL_MODE_MCP.

   MCPMODE=MANAGED\_MODE\_MCP
2. When a cluster/host wants to be monitored, it will configure its blueprint files that specify the master Actions Pro using Cluster name, Hostname, and Flag (indicating whether the host is to be monitored).
   1. In general, all hosts within this cluster should be registered as a whole. This is an example:

      MCP registration  
      rsmgmt.mcp.registration.enabled=true  
      rsmgmt.mcp.registration.password=<password>  
      rsmgmt.mcp.registration.port=8080  
      rsmgmt.mcp.registration.protocol=<http/https>  
      rsmgmt.mcp.registration.username=admin  
      rsmgmt.mcp.registration.uri=<CENTRAL MCP HOST NAME>
   2. A second method to register the host, the user can manually run a script to notify the central MCP. The RSConsole scripts for registration is located at:

      RSMGMT#/mcp>RegisterWithMCP.groovy
      If a system is set to be monitored, it should notify the central MCP each time it starts up, even though it could have been registered before. New registration of the existing host should not lead to duplicate records.
3. When a cluster/host wants to be removed from monitoring, to de-register the host, manually run a script to notify the central MCP. The rsconsole scripts deregistration is located at:

   RSMGMT#/mcp>DeregisterFromMCP.groovy
   The following conditions apply. When a cluster is:
   • No longer monitored, the local rules will take effect.
   • Being monitored, the global rules will take effect.

Create Rules

To access MCP:

1. From the Main Menu, select Report Administration > MCP.
2. Click New to create a set of threshold rules. The following criteria apply to rule creation.
   • Each rule will be evaluated against a single metric.
   • Users must be able to define a list of rules against a threshold. All rules must be met for the threshold to be effective.
   • When all conditions are met, the desired action will be executed.
   • All rules are auto deployed on all Actions Pro components by default.
3. Fill in the form as explained in the table below.

Field	Description
Module	This can be a group name or any name can entered. Allowed characters include letters, digits, single spaces, and underscores.
Name	This can be group option or any name can entered. Allowed characters include letters, digits, single spaces, and underscores.
Active	If a rule is set to inactive, it will not take effect. If it is set to active, it will take effect immediately
Severity	Rule severity options.
Actions	Rule action option to execute upon a breach.
Metric Group	Select a group category.
Metric Name	Each metric group has different menus selection depending on which Metric Group selected.
Metric Name	Each metric group has different menus selection depending on which Metric Group selected.
Source (optional)	Users can specify the metric source. If the source is not specified, all sources are matched. Source matching is a regular expression.
Conditions	Rule operation (=, >, <,>=, <=) against a fixed numeric value. A rule can also be defined as an operation ("contains") against a fixed string value. This is a required field.
Create	Creates a rule without a version.
Create with Version	Rules initial creation will NOT have a version, unless user explicitly choose to save a version.

Editing and Viewing

The following criteria apply to editing and viewing.

• A user can edit a rule with these conditions:
  • The rule must first be selected to edit it.
  • Users can edit and modify any field except MODULE and NAME.
• A user can create an additional version of a rule:
  • The user must save the rule using the "save as a version" option in order for the changes to take effect in alerts or deploy new changes.
  • Deployed rule version should not change.

note

The lower half of the display screen will remain blank unless the user selects an existing threshold rule.

See Create Rules for details on the available fields.

A rule can be copied. A copy will copy all versions of the rules. The latest revision is used for deployment.

For Delete:

• A rule for selected in order to delete it.
• A deleted rule should be "undeployed" from the monitored systems.
• "Delete a rule" will delete all versions of the rules.

Auto Deployed Rules

RULES by default are "Auto Deployed". The latest version of the rule is automatically deployed after the version is saved.

• When deploying a global rule, the rule should be deployed locally and to all other monitored clusters that do not have the latest version of this rule.
• A newly registered cluster will receive the latest version of all the "Auto-Deploy" rules from the master Actions Pro.
• All hosts under the cluster will be covered for deployment, including hosts added after the deployment.

Alerts

Open Main Menu > System Logs > Alert Log to view alerts.

The following conditions for Alerts apply:

• Alerts have fields to identify the host, component, severity, category, etc. that uniquely identify a problem.
• The duplicate alert will be grouped together with the ongoing (new or in progress) alert.
• Alerts can be suppressed but not eliminated.
• Users can always see the full alert list, or choose to see the filtered alerts list of any particular rule.

The alert dashboard should be periodically updated with the new alerts without the user manually refresh the page. Manage custom filter rules can be defined to filter and group alerts into different categories, and select only certain categories of alerts to show by default.

Act on Alerts

The following activities can be performed on an alert:

• Search alerts
• Sort alerts
• Start resolving, acknowledge, star, pin, and snooze if there is an email
• Send email notification
• Bring up guided procedures for an alert (if available)
• Connect to Resolution Routing with pre-populated fields

Alert Information

The following list is information that can be received or provided by alerts:

• System info including static info and load info such as CPU and Memory
• Health state (load, memory, thread count)
• Work statistics (runbook processed, current user, page accessed, etc.)
• Deployed gateway and their states
• Execution statistics
• Database state, connection count, transaction rate
• 3rd party software health state

Import Export

The following IMPEX conditions for rules apply:

• Threshold rules can be exported and imported.
• Imported rules will override existing rules with same name.

To import a rule:

1. Go to Main Menu > Import Export > Import Export Module.
2. Click New.
3. Under Components, click Add.
4. From the category drop-down list at the top-left, select System.
5. Select the Metric Threshold sub-category.

Appendix

Alert Fields

• alertid – globally unique sys_id. (Required)
• severity – Should have critical, severe, warning, info. Additional severity can be added without breaking existing system. (Required)
• summary - short summary of problem. No more than 64 characters long. (Required)
• detail - more detailed description of problem. No more than 512 characters long. (Optional)
• address - IP address or host name. (Required)
• source - additional source information GUID. If this alerts is triggered by other events, include the GUID of the source event. (Optional)
• component - rscontrol, rsview, etc. (Required)
• rid - Resolution ID or error code. This field generalizes a unique incident to a category of problem for deduplication purpose. It can be the Resolution ID if Resolution Routing is used, or a deterministic error code that duplicate messages can be grouped together. The source system is responsible for generating this ID. (Required)
• assignee - unassigned, <name>. (Required)
• status - New, Ack, InProgress, Closed. (Required)
• isSuppressed - true / false (Optional)
• suppressionRule – name of the suppression rule. If any rule has been used to suppress the alert, the name of the rule should be shown here. Only one rule is needed to suppress an alert. (Optional)
• groupRule - name of the group rule. If different kinds of alerts can be grouped together as a single alert, beyond the basic de-duplication rule, the name of the group rule will be recorded here. Multiple group rules can be applied to an alert. (Optional)
• firstOccurrence: TimesCTAp of the first occurrence of duplicate alerts. (Required)
• lastOccurrence: TimesCTAp of the last occurrence of duplicate alerts. (Required)
• count: Count of duplicate events. This field should be 1 or greater (Required)
• expiration: The validity of such alert, beyond that the alert is most likely invalid. For example, if the condition persists, a new alert would have been generated before expiration time. If a duplicate alert is received, the expiration time will be extended based on the latest message. (Required)
• suppressExpiration: This is used to temporarily suppress an alert, either because a rule says so or user chooses to snooze an alert. (Optional)

---

1. The official name of MCP is "Resolve Management Console". MCP originally stands for "Master Control Program" and could be re-interpreted as "Management Console Project".↩